[WIP] Change Event Based Governance Workflow#26758
[WIP] Change Event Based Governance Workflow#26758yan-3005 wants to merge 8 commits intoram/workflow-improvementsfrom
Conversation
…ted task nodes
Phase 1 of batch entity processing in governance workflows. All
automated task nodes (checkEntityAttributesTask, setEntityAttributeTask,
checkChangeDescriptionTask, rollbackEntityTask, sinkTask,
dataCompletenessTask) now process a List<String> of entity links via
entityList exclusively. The relatedEntity fallback in getEntityList()
is removed — batch nodes no longer have that path.
Key changes:
- PeriodicBatchEntityTrigger (singleExecutionMode=false): each child
process now receives global_entityList via ${entityToListMap[relatedEntity]}.
FetchEntitiesImpl pre-builds entityToListMap (entity -> List.of(entity))
so the JUEL expression resolves without static class references.
- Batch node impls (6 files): removed relatedEntity fallback from
getEntityList() and the now-unused RELATED_ENTITY_VARIABLE import.
entityList is the only input path.
- BPMN builders: putIfAbsent(ENTITY_LIST_VARIABLE, GLOBAL_NAMESPACE) for
all batch-capable nodes; relatedEntity is never added to inputNamespaceMap
by builders.
- v1130 migration (addEntityListToNamespaceMap): updated to also strip
relatedEntity from batch node inputNamespaceMaps, covering both fresh
upgrades (add entityList + remove relatedEntity) and instances that
already ran the previous migration (entityList present, remove relatedEntity).
Migration remains idempotent.
- GlossaryApprovalWorkflow.json: removed relatedEntity from all 13 batch
node inputNamespaceMaps. userApprovalTask nodes keep relatedEntity.
- JSON schemas: entityList added to trigger output and batch node
inputNamespaceMap/input definitions.
- Integration tests: updated to reflect entityList-first structure across
all batch-capable workflow node configurations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
...resources/json/schema/governance/workflows/elements/triggers/periodicBatchEntityTrigger.json
Outdated
Show resolved
Hide resolved
...ice/src/main/java/org/openmetadata/service/governance/workflows/elements/TriggerFactory.java
Outdated
Show resolved
Hide resolved
| private String resolveEntityType(DelegateExecution execution) { | ||
| String processDefinitionKey = execution.getProcessDefinitionId().split(":")[0]; | ||
| if (processDefinitionKey.contains("-")) { | ||
| String[] parts = processDefinitionKey.split("-"); | ||
| if (parts.length >= 2) { | ||
| return parts[parts.length - 1]; | ||
| } | ||
| } | ||
| return (String) entityTypesExpr.getValue(execution); | ||
| } |
There was a problem hiding this comment.
💡 Bug: resolveEntityType falls back to expression on ambiguous key
In FetchChangeEventsImpl.resolveEntityType() (lines 136-145), the process definition key is split on - and the last segment is used as the entity type. If the process key contains - but has fewer than 2 parts after splitting (which can't actually happen with split), or if the entity type itself contains a -, the wrong value will be extracted. For example, an entity type like api-endpoint would be parsed as just endpoint.
Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion
…ts-change-event-offset - Keep two-phase migration in MigrationUtil (raw JSON Phase 1, BPMN redeploy Phase 2) - Keep workflowFqn parameter in PeriodicBatchEntityTrigger constructor - Keep FetchChangeEventsImpl as primary fetch task (no FetchEntitiesImpl) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Combines both changes: - hasBatchModeNodes(workflow) for singleExecutionMode (from base) - workflow.getFullyQualifiedName() for workflowFqn (from PR branch) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
…ts-change-event-offset
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
…ts-change-event-offset
yan-3005
added
the
safe to test
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
…ts-change-event-offset
Code Review 👍 Approved with suggestions 2 resolved / 3 findingsRefactors governance workflows to process change events with batch entity handling for automated task nodes. Resolved schema validation and batch sink execution issues; consider handling ambiguous entity type resolution more explicitly in FetchChangeEventsImpl. 💡 Bug: resolveEntityType falls back to expression on ambiguous keyIn ✅ 2 resolved✅ Bug: Schema maxItems:2 rejects 3-element output arrays used in tests
✅ Bug: singleExecutionMode hardcoded to false, batch sink broken
🤖 Prompt for agentsOptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
Jest test CoverageUI tests summary
|
|
|
🟡 Playwright Results — all passed (18 flaky)✅ 3398 passed · ❌ 0 failed · 🟡 18 flaky · ⏭️ 216 skipped
🟡 18 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
1 participant
Describe your changes:
Fixes https://github.com/open-metadata/openmetadata-collate/issues/3364
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>Summary by Gitar
FetchEntitiesImplwithFetchChangeEventsImplfor incremental offset-based change event processingCommitChangeEventOffsetImplto persist consumed offsets per workflow and entity typechange_event(entityType, offset)for efficient filteringchange_event_consumers.idfrom VARCHAR(36) to VARCHAR(768) for workflow consumer IDsPeriodicBatchEntityTriggerto fetch change events instead of searching; added workflow FQN parameterfiltersoptionalWorkflowDefinitionResourceITwith multi-batch change event test flowThis will update automatically on new commits.